A hybrid MCDM model combining Fuzzy-Delphi, AEW, BWM, and MARCOS for digital economy development comprehensive evaluation of 31 provincial level regions in China

With the deepening of a new round of technological revolution and industrial reform, digital technology has been continuously innovated and widely penetrated into various economic fields. The digital economy (DE) is gradually becoming the focus of China’s economic development planning and a new engine to enhance national strength. Evaluating the development level of DE in various regions is conducive to timely discover the shortcomings in China’s DE development, as well as provide an important basis for putting forward corresponding policy suggestions. This investigation established a hybrid multi-criteria decision making (MCDM) model for evaluating DE development of 31 provincial level regions in China ranging from 2015 to 2020. Firstly, the evaluation indicator system is established from digital infrastructure, integrated development, social benefits, innovation ability, and electronic-commerce dimensions containing 17 quantitative sub-criteria based on Fuzzy-Delphi method. Secondly, integrated weights of 17 sub-criteria from 2015 to 2020 are computed in terms of objective weights calculated by the anti-entropy weight (AEW) approach from 2015 to 2020 and subjective weights obtained via the best-worst method (BWM). Thirdly, MARCOS model is applied to evaluate the DE development degree of various regions in China ranging from 2015 to 2020. Case analysis illustrates that the DE development of Guangdong, Jiangsu, Zhejiang, and Beijing always maintain in the top four from 2015 to 2020, while the southwest and northwest regions in China are obviously fall behind others. And the DE development degree of various regions is primarily affected under the integrated development performance, innovation ability performance, and social benefits performance. Therefore, the backward regions should emphasize the development of software industry and information technology industry. The robustness of the proposed MCDM model combining Fuzzy-Delphi, AEW, BWM and MARCOS is discussed employing three similarity coefficients of rankings. And it is verified that the proposed MCDM model has superior robustness and validity in evaluating DE development.

1 Introduction related statistical data. Through empirical analysis and comparison discussion, diversities in the development level of DE among various provinces, autonomous areas, and megacities can be discovered, so that local governments can formulate targeted policies and measures to improve the development level of DE.
The primary contributions of our research are as below. Firstly, the comprehensive evaluation indicator system is built from digital infrastructure, integrated development, social benefits, innovation ability, and electronic-commerce dimensions containing 17 sub-criteria based on Fuzzy-Delphi approach taking the knowledge and judgments of experts into consideration.
Secondly, the subjective weights are identified applying the BWM and the objective weights are computed via the AEW method. The integrated weights are obtained via the basic principle of moments estimation by calculating the subjective and objective weights coupling coefficients. Thus, the actual data information and the attributions of sub-criteria themselves can be sufficiently considered so that the reliability of sub-criteria weights can be improved.
Thirdly, a new MCDM method named MARCOS, will be employed to estimate the development level of DE in 22 provinces, 5 autonomous areas, and 4 provincial level megacities in China, which can enrich the methodology of digital economy development evaluation.
Fourthly, the robustness of the established MCDM model consisting of the Fuzzy-Delphi method, BWM, AEW method, and the MARCOS model is verified. Spearman's rank correlation coefficient (r S ), weighted Spearman's rank correlation coefficient (r W ), and WS similarity coefficient (WS) are employed to check the robustness of provincial level regions rankings obtained by the established MCDM model and four comparison models.
The rest sections of this investigation are designed as below. Previous literature is reviewed in Section 2. Section 3 introduces the primary methodologies applied to build the hybrid MCDM model for DE development evaluation. Section 4 describes the process of establishing the index system for comprehensive evaluation of DE development and introduces the sub-criteria contained in final index system. Case analysis is conducted in Section 5. The robustness of the established hybrid MCDM model is verified in Section 6. Section 7 draws conclusions.

Literature review
To evaluate the development level of DE, it is imperative to build a comprehensive evaluation indicator system which can accurately and comprehensively describe the characteristics of DE and elements required for DE development. In 2014, the Organization for Economic Cooperation and Development (OECD) released the "Measuring the Digital Economy-A New Perspective" report, which constructs the evaluation index system of DE development from perspectives of intelligent infrastructure, social application, innovation ability, as well as growth and employment, including 38 indicators [5]. In 2016, aiming at evaluating the performance of European DE development level and track the evolution of digital competitiveness of European Union (EU) countries, the EU issued "Digital Economy and Society Index (DESI) in 2016". The EU DE and society index system constructs the evaluation indicator system of DE development from five perspectives of network connection, human resources, network application, digital technology integration and digital public service, including more than 30 indicators. The indicator selection has a high theoretical level and certain rationality, however, a relatively simple weighted sum was used as the final result when calculating the DE development index [6]. In 2018, the U.S. Bureau of Economic Analysis issued "Defining and Measuring Digital Economy" report, which proposed that the DE is composed of three aspects containing DE infrastructure, electronic-commerce and digital media and constructed price and quantity indicators to measure DE gross production and added value according to these three aspects. The index system, containing 12 indicators, highlights the important position of digitization in economic growth, but ignores the great value that digitization brings to government affairs, people's livelihood, environmental protection and other industries [7]. In 2017, the Digital China Research Institute of the National Information Center put forward the "five in one" index system to measure the DE in the report of "Digital China Construction Communication". This index system includes technical capacity, core development and guarantee level dimensions. And the number of primary, secondary and tertiary indicators contained in the index system are 3, 12 and 37, respectively. The index system is relatively comprehensive, but involves more qualitative indicators, of which the preference values are difficult to obtain, and does not consider the benefits of emerging technologies for the DE [8]. In 2017, Chinese Academy of Information and Communications issued the "White Paper on the Development of China's Digital Economy", which gives the evaluation index system of China's DE index composed of three parts. The first part is the leading index, which is composed of eight indicators. The second part is the consistency index, which is composed of 10 indicators. The third part is the lag index, which is composed of four indicators. Compared with the previous index system, this index system adds lagging indicators, and considers the digital economic growth brought by new digital technology, but does not consider the risk factors brought by the development of DE and the influencing factors of people's livelihood such as electronic-government and electronic-medicine [9]. In 2017, the Shanghai Academy of Social Sciences issued the "Blue Book on Digital Economy", which constructed the evaluation indicator system of global DE competitiveness from the perspectives of infrastructure, industry, innovation and governance, and analyzed the DE competitiveness of major countries in the world [10]. In September 2018, the Digital Economy Research Institution and other institutions jointly issued the "Global Digital Economy Development Index in 2018", which constructs a DE development indicator system from digital infrastructure, digital consumers, digital business ecology, digital public services, as well as digital education and scientific research perspectives, including 16 indicators. This index system not only evaluates the development level of DE, but also evaluates the industrial structure and development path of DE [11]. Zhang et al. investigates the overall development of China's DE from the perspectives of infrastructure, primary and advanced applications of information and communication technology, enterprise digital development as well as the development of information and communication technology industry. The research shows that the development of China's DE shows an upward tendency, but the increase rate shows a slowing trend. However, the index system greatly highlighted the economic benefits and lacks the measurement of social benefits, and does not consider the impact of network security level on the development of DE [12].
For methods employed in measuring the DE development, Chen comprehensively used entropy method, Dagum-Gini coefficient method and panel Tobit model to explore the development level, regional diversities and driving factors of DE in various provinces and cities utilizing the panel data of 30 provinces and cities ranging from 2013 to 2019. Results showed that from the perspective of driving factors, the level of economic development, industrial structure, human capital and scientific and technical development could strongly promote the development of DE, while government intervention will inhibit the development of DE. Among them, scientific and technical development and intellectual property protection have a great influence on the DE development [13]. Li and Han constructed the quantitative indicator system of DE development level from digital infrastructure, digital industrialization and industrial digitization combined with the connotation and extension of DE. Then the development tendency of DE in China from 2010 to 2018 was measured based on entropy method, and a grey prediction approach was constructed to forecast the development tendency of DE from 2019 to 2028 [14]. Dong constructed the index system to evaluate the DE of China from digital infrastructure, integrated development, innovation ability, social benefits and electronic-commerce. Then the scores of DE development level of all provinces in China from 2015 to 2019 were calculated based on PCA-EM model and analyzed the regional development differences and the causes of the differences with the help of coefficient of variation analysis and geographic detectors [15]. Li employed DEA model to comprehensively evaluate and analyze the development efficiency of China's DE from the static and dynamic aspects utilizing the constructed evaluation indicator system of DE development efficiency. Then, in accordance with relevant economic theories, the basic assumptions affecting the development efficiency of DE are put forward, and an empirical test is conducted by using Tobit model [16]. Li constructed a comprehensive, scientific, and complete comprehensive evaluation indicator system of DE development level from the perspectives of basic resources, integrated development, innovation ability, social benefits, and network security. Then, taking Jiangsu Province as an example, the DE development level of 13 cities in Jiangsu Province from 2014 to 2017 was calculated by using entropy weight approach and approximate ideal solution ranking model and a comparative analysis was also conducted on the overall level and five dimensions of DE development in various cities [17]. Chen et al. employed nighttime light remote sensing data to estimate the DE and utilized the Zipf's law to evaluate the DE growth from the city level [18]. Nela Milošević et al. employed the Composite I-distance Indicator (CIDI) methodology to evaluate as well as rank digital performance of 28 countries in European Union [19].
On the basis of the analysis of existing literature, it can be found that firstly foreign research on DE started earlier, and most of the index systems have been tested for a long time with high credibility and authority. Domestic research on DE started relatively late, but the evaluation indicator system of DE development is more comprehensive. Secondly, it is difficult to find literatures on the comprehensive evaluation of DE development employing MCDM model, while traditional researches on the measurement of DE development primarily depended on the statistical data [20,21]. Nevertheless, the coherence and comparable objective data are performed infrequently and hard to be collected, as it needs a great amount of labor work and not all the objective data are usable for various areas. Owing to the data restriction, the development of DE is hard to be analyzed. Thirdly, most of the previous investigations are on the basis of the national level, which would neglect the differences between regions. Considering that all provincial level regions in China have their own DE development strategies and plans, evaluating the DE development level of each provincial level region can provide more policy support for the DE development of the relevant provincial level region. Therefore, this paper evaluates and analyzes the development level of provincial DE in China, and puts forward targeted suggestions combined with the development characteristics of provincial DE, so as to accelerate the high-level and high-quality development of China's DE.
The index system for evaluating the DE development is established based on Fuzzy-Delphi considering about experts knowledge and experience. And the BWM is employed to determine the subjective weights, as it only needs to compare the importance of each sub-criterion with the best and the worst sub-criteria, which is time-saving and convenient based on experts' judgments. And the AEW method is utilized to determine the objective weights based on objective data. These two methods have been employed in many fields to determine sub-criteria weights in comprehensive evaluation, such as aircraft selection [22], green supplier selection [23], and enterprise debt decision [24]. To comprehensively considering the importance of experts judgments and objective data, a weight integrating method is employed to combining subjective weights and objective weights based on the basic principle of moments estimation. Then integrated weights can be obtained. Afterwards, a new MCDM method named MARCOS, proposed by Željko Steviā and Dragan Pamučar in 2020, will be employed to estimate the development level of DE in 22 provinces, 5 autonomous areas, and 4 provincial level megacities in China. MARCOS model can take the positive and negative ideal solutions into account at the same time, and rank the provincial level regions based on the utility functions, which can make the results have superior robustness and accuracy [25,26]. Therefore, a MCDM framework combining Fuzzy-Delphi, the BWM, the AEW, and MARCOS methods is established for DE development comprehensive evaluation of 31 provincial level regions in China.

Methods
The hybrid MCDM model for comprehensive evaluation of DE development is systematic and rigorous integrating the Fuzzy-Delphi method, BWM, AEW method and MARCOS model. The concrete process is illustrated as Fig 1.

Fuzzy-Delphi method
Delphi method has been extensively applied in decision making researches, which can be used to collect the most valid opinions from experts [27]. In traditional Delphi method, experts can receive feedback and reconsider their opinions provided previously via four rounds consultation [28], of which the process is superfluous and hard to gather consistent judgments via several rounds consulting. Hence, to deal with the above demerits, the Fuzzy-Delphi method, combining traditional Delphi method and fuzzy theory, has been established. In this method, experts should make three points judgment using triangular fuzzy numbers (TFNs) [29]. Membership degree functions (MDFs) are utilized to indicate the judgments of experts. Advantages of MDFs are that experts do not need to adjust their judgments through several rounds consultation and MDFs can sufficiently use all judgments. Due to the superiority of the Fuzzy-Delphi, many researches utilized it to construct the evaluation indicator system [30]. The steps of Fuzzy-Delphi method in detail are described as below.
Step 1: By gathering the judgments of experts, we can identify the upper and bottom bounds of confidence interval [0, 10] for each indicator. If the value is larger, the indicator is more critical. The upper bound represents the most positive value, while the bottom one implies the most conservative value.
Step Step 3: Examining the consistency of experts' opinions and identifying the coherent value G i of indicator i [30,31].
(1) If C i U < P i L , the judgment for indicator i are coherent, and G i is determined by:

PLOS ONE
While if the grey scope value Z i ¼ C i U À P i L is larger than M i ¼ P i U À C i M , the opinions of experts will be conflicting. Then we should process the Steps 1 and 2 again until the opinions are coincident. G i demonstrates the consistent degree identified by experts judging the important degree of each indicator. The larger the value of G i , the more important the indicator will be. To find an optimal value to verify the important degree of each indicator, the geometric mean value of the upper and bottom bounds of all indicators is calculated, deeming as the confident value to screen final indicators.

Sub-criteria weighting method on the basis of AEW and BWM
(1) AEW method Supposing that there exist m probable situations in the system, and every situation occurs with a probability of p j (j = 1,2,. . .,m), then the entropy is calculated as [32]: where 0�p j �1, and ∑ j p j = 1. For AEW approach, assuming the amount of alternatives is m and the amount of sub-criteria is n, the value of sub-criteria i with regard to alternative j is x ij (i = 1,2,. . .,n, j = 1,2,. . .,m), hence, the assessment matrix is described as X = [x ij ] n×m . Thus, the anti-entropy value of every sub-criterion can be obtained via [33]: where r ij = x ij /∑ j x ij . Via standardizing the anti-entropy value, the objective weight w 1i of every sub-criterion is calculated by: (2) The BWM The BWM is a pair comparison-based approach, the fundamental theory of which is similar with AHP, but simpler than that. AHP method determines weights via pair-wise comparisons after nðnÀ 1Þ 2 comparisons (supposing that there are n sub-criteria). Nevertheless, the BWM only needs 2n-3 comparisons, 1 for comparing the best sub-criterion with the worst one, n-2 for comparing the best sub-criterion with others, and n-2 for comparing the others with the worst one [34,35]. The concrete process of the BWM is introduced as below.
Step 1: Choose the best and the worst sub-criteria from the constructed final sub-criterion system {C 1 , C 2 ,. . .,C n }. The best sub-criterion is the most critical and desirable one which can accurately describe the distinguishing features of the alternatives, nevertheless, the worst one is contrary.
Step 2: Compare the best one with others using numbers from 1 to 9. The larger value indicates the best one is much more crucial than others. Comparing results are written as: where a Bi implies the critical degree of the best one B over indicator i, and a BB = 1.
Step 3: Compare other indicators with the worst one using numbers from 1 to 9. Results are: where a iW represents the importance degree of indicator i over the worst one W, and a WW = 1.
Step 4: Compute the optimal weights ðw * 1 ; w * 2 ; . . . ; w * n Þ of all indicators. To obtain the optimal weight of every sub-criterion, we should minimize the maximum differences fjw B À a Bi w i j; jw i À a iW w W jg for all sub-criteria, and the formula is linearized referred to Ref [36]: min ε s:t: Afterwards, the optimal weights ðw * 1 ; w * 2 ; . . . ; w * n Þ are determined in terms of the above formula. Hence, subjective weights written as w 2i of sub-criterion i are identified.
Step 5: Testify the comparing consistence by coherence test. The greatest value of ε is calculated through processing the Formula (8). Table 1 illustrates the coherent indicator which are selected based on the values of a BW ranging from 1 to 9. Hence, the consistent ratio can be determined via: The smaller the consistent ratio is, the more consistent the comparing will be.
(3) Integrated weights coupling subjective weights and objective weights The BWM, applied to determine subjective weights, makes a significance ranking of subcriteria in terms of experts' knowledge and experience. However, it can only reflect the subjective opinion of experts about the critical degree of sub-criteria. The AEW method, applied to calculate objective weights, makes a significance ranking of sub-criteria in terms of the objective data of each sub-criteria in regard to alternatives. However, it is difficult for subjective weights to reflect actual data information of sub-criteria, while objective weights can reflect actual data information but will change with actual data change of alternatives. Thus, the volatility of objective weights is stronger than subjective ones. Taking the merits and demerits of subjective and objective weights into consideration, recent researches focused on weights combination methods [37][38][39]. Integrated methods combining subjective and objective weights cannot only avoid the issue that objective weights emphasize actual data and ignore the subcriteria attributes bringing about the unreasonable weights results, but also avoid the issue that experts' opinions are too subjective, so as to obtain more effective weights.
In this investigation, integrated weights of subjective and objective weights are estimated in terms of the basic principle of moment estimation. Owing to the difference of the relative significance of subjective and objective weights for sub-criteria, subjective and objective weight coupling coefficients are obtained via:

PLOS ONE
where δ i and ε i are the objective and subjective weight coupling coefficients of sub-criterion i, respectively. Afterwards, the integrated weight θ i of sub-criterion i integrating subjective and objective weights of sub-criterion i is computed via:

Comprehensive evaluation model based on MARCOS
Measurement of alternatives and ranking in accordance with compromise solution (MAR-COS), proposed by Željko Steviā and Dragan Pamučar in 2020 [40], will be applied to estimate the development of the DE of 31 provinces in China. In MARCOS method, through comparing the reference values of alternatives with ideal values, the utility functions of alternatives are determined, and the compromise ranking related to the ideal alternative and the negative ideal alternative is attained. MARCOS method can consider positive and negative ideal solutions simultaneously, and consider the probability of a great amount of standards and alternatives while keeping the steadiness of the model, so as to obtain the results with good robustness and precision. The concrete procedure is expressed as below.
Step 1: Establishing extended original matrix X = [x ij ] n×m . The positive ideal solution (AI) and negative ideal solution (AAI) are defined to expand the matrix X = [x ij ] n×m . The AI is the alternative with the best characteristics, while the AAI is contrary.
Among which, in accordance with the attribute of the sub-criterion, AAI and AI are determined by where B represents maximum type sub-criteria and C indicates minimum type sub-criteria.
Step 2: Standardizing extended initial matrix N = [n ij ] n×m . The standardized decision matrix is attained by normalizing the matrix X = [x ij ] n×m .
where x ij and x aj are the elements in initial matrix X = [x ij ] n×m .
Step 3: Step 4: Calculating the utility degree K j of alternative j. The utility degrees K j relative to the negative and positive ideal solutions are computed via the following formula: where S(j = 1,2,. . .m) represents the sum of elements in the weighting matrix V.
Step 5: Identify the utility function f(K j ) of the alternative j. The concrete expression is: where f ðK j À Þ and f ðK j þ Þ imply the utility functions relative to the negative and positive ideal solutions. They can be expressed as below.
Step 6: Ranking alternatives in accordance with the final value of the utility function. The ideal alternative is the one with the greatest utility function value.

The procedure of the established hybrid MCDM model for comprehensive evaluation of DE development
The detailed procedure employing the hybrid MCDM model for comprehensive evaluation of DE development of 31 provincial level regions in China is introduced as below and Fig 1. Step 1: Screen the crucial indicators to construct the final comprehensive evaluation indicator system employing Fuzzy-Delphi method. Firstly, we need to collect and summarize initial sub-criteria from related reports and researches on DE development. Secondly, 5 authoritative experts in DE development research field are invited to compose the decision group. Thirdly, experts should judge the important degree of each indicator using the Fuzzy-Delphi method so that we can screen the critical sub-criteria in terms of Eqs (1) and (2). Afterwards, the final indicator system for comprehensive evaluation of DE development can be constructed.
Step 2: Collect the objective data for all sub-criteria to obtain the decision matrix X = [x ij ] n×m . Considering that all sub-criteria contained in final index system are objective sub-criteria, so that actual data can be collected from statistical yearbook of each provincial level region and the official website of National Bureau of Statistics. Afterwards, the decision matrix X = [x ij ] n×m are obtained.
Step 3: Compute the comprehensive weights of all indicators coupling the subjective weights obtained via the BWM and objective weights obtained via the AEW. Firstly, objective weights are calculated on the basis of the decision matrix utilizing the AEW method through Eqs (4) and (5). Secondly, subjective weights are judged via the BWM taking experts' judgments into consideration through Eqs (6)- (8). Finally, comprehensive weights integrating objective and subjective weights are obtained through Eqs (10)-(11).
Step 4: Obtain weighted normalized decision matrix V = [v ij ] n×m . Firstly, extended initial matrix should be established combining the AAI and AI data series which can be attained through Eqs (13) and (14). Secondly, the standardized extended initial matrix N = [n ij ] n×m can be obtained through Eqs (15) and (16). Finally, the weighted standardized decision matrix V = [v ij ] n×m can be calculated via Eq (17).
Step 5: Determine the utility function f(K j ) of the provincial level region j. Firstly, we need to compute the utility degrees of the provincial level regions relative to the negative and positive ideal solutions of the provincial level region j using Eqs (18) and (19). Then the utility function f(K j ) of the provincial level region j can be determined via Equs (21)-(23).
Step 6: Rank provincial level regions in accordance with the final value of the utility function f(K j ) and select the optimal provincial level region with the greatest utility function value, which represents the DE development level of this provincial level region is superior than others.

Indicator system of comprehensive evaluation on DE development
To comprehensively and accurately evaluate the development of DE of various provinces in China, the construction of comprehensive evaluation index system plays a significant role. Firstly, the initial indicator system is built from digital infrastructure, integrated development, social benefits, innovation ability, and electronic-commerce dimensions containing 31 initial sub-criteria through referring to related reports and published literature. Secondly, we utilized Fuzzy-Delphi to select the crucial indicators and construct the final index system considering 5 experts judgments, who are invited from universities, governments, and DE research institute.
During the process of screening the critical sub-criteria, experts firstly should judge the important degree of each initial indicator and the most positive and the most conservative values of each indicator are determined. Secondly, the conservative TFN ðC i L ; C i M ; C i U Þ and positive TFN ðP i L ; P i M ; P i U Þ of each sub-criterion are computed. Thirdly, the coherence of experts' judgments can be assessed via computing the values of G i using Eqs (1) and (2). Afterwards, the critical sub-criteria are screened via comparing values of G i with the confident value (which is 7.26 computed by the geometric average value of upper and bottom bounds of all original indicators). Results for the process of indicators screening are illustrated in Table 2, and 17 critical indicators are chosen to compose the final indicator system for DE development evaluation of 31 provincial level regions in China.
The final index system for DE development evaluation is illustrated in Fig 2, which is composed of 17 sub-criteria attributed to five aspects of digital infrastructure, integrated development, social benefits, innovation ability, and electronic-commerce. All the final sub-criteria are all qualitative and benefit-type.
For digital infrastructure, DE infrastructure is a crucial part to ensure the steady operation and development of DE. All activities of DE are upon infrastructure construction. The construction of DE infrastructure is reflected in the popularity of information techniques hardware and internet. Hence, based on experts' judgments, telephone penetration ratio, the number of internet broadband access users, and long-distance optical cable line length are selected. Telephone penetration ratio refers to the popularity of telephones in a region, measured by the value of all telephones in 100 people. It can reflect the degree of mobile network construction and the application of mobile network in the region. The number of internet broadband access users indicates the coverage of broadband network, which can reflect the construction of network from the perspective of broadband. The length of long-distance optical cable line can reflect the coverage of long-distance optical cable line in a certain area and the application of new internet technology.
For integrated development, the development of DE is not independent, but complementary to the traditional economy. With the help of information technology, DE innovates the

PLOS ONE
production technology and operation mode of traditional economy. In the meantime, DE also excavates more effective information for traditional economic system, so as to make more effective decisions. Therefore, this paper selects the number of websites per 100 enterprises owned, software business income, express business income, total post and telecommunications business, and income from information technology services as secondary indicators to

Case analysis
After screening the critical sub-criteria employing the Fuzzy-Delphi method, the evaluation indicator system for DE development can be established. Considering that all sub-criteria are quantitative, actual data of all sub-criteria with regard to 31regions in China can be gathered from the website of National Bureau of Statistics (http://www.stats.gov.cn/). Aiming at comparatively researching on the development level of DE in 31 regions in different years, data from 2015 to 2020 are collected, which are listed in S1 Data. Based on the collected data, case analysis can be processed.

Determine the integrated weights of all sub-criteria
(1) Determine the objective weights employing AEW method Based on the collected actual data of all indicators with regard to 31 regions in China from 2015 to 2020, objective weights of all sub-criteria from 2015 to 2020 are calculated via Eqs (4) and (5). Considering that the actual data are various in different years, so that the objective weights are various in various years. Results of objective weights from 2015 to 2020 are listed in Table 3.
(2) Determine the subjective weights employing the BWM Subjective weights are obtained via the BWM taking the opinions and judgments given by 5 selected experts. Since subjective weights only considers experts' opinions considering about their knowledge and experience without taking the actual data of indicators into account with regard to provincial level regions, hence, final subjective weights has only one set of data regardless of years. Firstly, the best and the worst indicators are chosen from the final constructed indicator system. The best indicator is the one with the great significance to describe the particular characteristics of DE development, while the worst one is contrary. After collecting the judgments of 5 experts, the best and the worst indicators chosen by 5 experts are listed in Table 4.
Secondly, 5 experts should compare the important degree of the best indicator to others and others to the worst indicator using a score ranging in the interval of [1,9]. The value of 1

PLOS ONE
demonstrates an equal important degree between the best indicator and others, while the value of 9 means the best one is extremely more critical and desirable than others. Values increasing from 1 to 9 indicate the crucial and desirable degree increase progressively. Results of the significance comparison for the best one with others are illustrated in Table 5 and results of the important degree comparison for others with the worst one are described in Table 6.
Thirdly, in terms of the comparison results above, subjective weights of all indicators are calculated employing a linear model shown in Eq (8). Results of subjective weights are demonstrated in Table 7, and afterwards an average value of weight for each indicator is calculated to attain the final subjective weight of each indicator. Results of the coherence indicator ξ* judged by 5 experts are also listed in Table 7. And the consistent ratio of 5 experts' judgment would be calculated in terms of Table 1 and Eq (9). It can be clearly found that the comparisons are of highly consistency as the consistent ratio are close to zero.

(3) Determine the integrated weights
Based on objective weights obtained via the AEW method from 2015 to 2020 and subjective weights obtained via the BWM, the integrated weights from 2015 to 2020 combining subjective and objective weights are obtained in accordance with the basic principle of moment estimation, which are computed by Eqs (10) and (11). The integrated weights regard of different years are shown in Table 8. The top four indicators with relatively greater integrated weights are C 42 with 0.0839 in 2020, C 31 representing express quantity with 0.0838 in 2020, C 23 representing express business income with 0.0746 in 2020, and C 41 with 0.0745 in 2020. And the last three indicators with relatively less integrated weights are C 13 representing long distance optical cable line length with 0.0406 in 2020, C 11 representing telephone penetration ratio with 0.0391 in 2020, and C 32 representing average wage of urban employees in information transmission, computer services and software industries with 0.0320 in 2020.
The comparison of integrated weights on criteria level in different years is depicted in Fig 3. It is illustrated that the integrated weights of C 2 on behalf of integrated development are all Table 4. The best and the worst indicator chosen by 5 experts.

The best indicator
The worst indicator 1 The number of internet broadband access users (C 12 ) Electronic-commerce purchase amount (C 52 )

2
The number of websites per 100 enterprises owned (C 21 ) The proportion of enterprises with electronic-commerce (C 53 )

3
Research & development funds for industrial enterprises above designated size (C 42 ) Express quantity (C 31 )

4
Research & development funds for industrial enterprises above designated size (C 42 ) The proportion of enterprises with electronic-commerce (C 53 )

5
The proportion of enterprises with electronic-commerce (C 53 ) Express quantity (C 31 ) https://doi.org/10.1371/journal.pone.0283655.t004 largest in different years, implying that the integrated development of informatization and industrialization is of great significance in DE development. The integrated weights of C 4 representing innovation ability ranks the second, implying that the number of patent applications

PLOS ONE
increasingly mature, the significance of social benefits in the development evaluation of DE is becoming increasingly prominent. The difference of objective weights obtained via the AEW method from the year of 2015 to 2020 and subjective weights obtained via the BWM on indicators level are illustrated in Fig 4. We can find that there exists little difference in the objective weights of various years calculated utilizing actual data using the AEW method, while subjective weights obtained based on experts' opinions and judgments using the BWM are quite different from the objective weights of various years. Taking various weights of C 42 as an example, the subjective weight of C 42 is 0.1078 which is the greatest subjective weights, while the objective weights from 2015 to 2020 are 0.0670, 0.0656, 0.0657, 0.0646, 0.0628, and 0.0621, respectively, of which the objective weights are at the midstream level among all objective weights computed. Thus, the above comparison demonstrates that there exist significant differences between objective weights and subjective weights for the same indicator. Therefore, integrated weights combing subjective and objective weights should be calculated to assess the development of DE to avoid the bias in evaluation results via utilizing single subjective weights or objective weights.

Results of DE development evaluation for 31 provincial level regions in China
According to the objective data of indicators with regard to 31 regions in China from 2015 to 2020, the normalization matrix can be obtained via Eqs (15) and (16). Through combining with the integrated weights, the weighted normalization decision matrix is attained via the Eq (17). Afterwards, the utility function values f(K j ) of 31 provincial level regions can be calculated based on Eqs (18)- (23). The utility function values of 31 regions from 2015 to 2020 are illustrated in Fig 5. As is demonstrated in Fig 5, the values of utility function in Guangdong, Jiangsu, Zhejiang, Beijing, Shandong, Shanghai, and Sichuan are much greater than other provincial level regions, which are all greater than 0.30 from 2015 to 2020. Among them, Guangdong, Jiangsu,

PLOS ONE
Zhejiang, and Beijing always maintain in the top four. Guangdong province has conscientiously implemented the planning requirements of "the implementation plan of the national pilot zone for innovative development of DE", and Shenzhen, known as the scientific and technological innovation center located in Guangdong province, has congregated a large amount of high-tech enterprises, communication enterprises, and software enterprises which can greatly accelerate the development of DE, hence, software business income, income from information technology services, employment of urban units in information transmission, software and information technology services, and the number of patent applications and research & development funds of industrial enterprises above designated size are much higher than other regions. Therefore, the development level of DE in Guangdong takes the lead from 2015 to 2020 with the average value of utility function higher than 0.90.
As an experimental field for building a smart digital city, the eastern coastal areas make sufficient use of the talent, technology, infrastructure and other resources brought by economically developed areas to vigorously carry out digital construction. Hence, the development level of DE of Jiangsu and Zhejiang ranked the second and the third, respectively, of which the average values of utility functions from 2015 to 2020 are 0.6398 and 0.5596, respectively. As the political and economic center, Beijing attaches importance to the investment of knowledge-based talents and owns high telephone penetration ratio, software business income, information technology service income, and relatively higher proportion of enterprises with electronic-commerce transactions. Beijing also highlights the appliance of big data, cloud computing, AI, and other techniques in decision-making and economic activities, which can

PLOS ONE
industries, electronic-commerce sales amount, and the proportion of enterprises with electronic-commerce transactions in Shanghai are much higher than Shandong in 2019 and 2020, so that Shanghai surpassed Shandong ranked the fifth in 2019 and 2020.
Due to the late economic development in the southwest and northwest regions in China, the relevant talents, technology and basic facilities are still accumulated and constructed, and so that the number of internet broadband access users, the employment of urban units in information transmission, software and information technology services, the number of patent applications and research & development funds of industrial enterprises above designated size, and the number of research & development personnel in enterprises above designated size are much lower than other regions. Thus, the development level of the DE is obviously lower than others. Among them, Xinjiang and Ningxia ranked at the last two positions, of which the average values of utility function from 2015 to 2020 are 0.1218 and 0.1154, respectively. The development level of DE in the northeast region (especially Jilin and Heilongjiang provinces) rank at the downstream position of China. The primary causes are that the economy growth in the northeast region primarily relies on the heavy old industry and the primary industry and the industrial structure is relatively backward and the technology is relatively low. Hence, the number of websites per 100 enterprises owned, software business income, income from information technology services, research & development funds for industrial enterprises above designated size, and the number of research & development personnel in enterprises above designated size are relatively lower than other regions, so that the DE development level of these regions are inferior to others. Fig 6 illustrates the detailed performance of 31 regions from 2015 to 2020, including digital infrastructure, integrated development, social benefits, innovation ability, and electronic-commerce performances.
(1) For digital infrastructure performance, generally, Guangdong performs the best followed by Sichuan and Shandong. The development of digital infrastructure in Sichuan province demonstrates an upward tendency year by year, and the value of utility function of digital infrastructure of Sichuan surpassed that of Guangdong in 2019 ranking the first. Consistent with the overall ranking, the digital infrastructure performance of the southwest and northwest regions in China still scores near the bottom, especially Ningxia province. Through comparing the utility function values of digital infrastructure performance in different years in 31 provincial level regions, it can be obviously discovered that the construction of digital basic facilities in different provincial level regions has been strengthened year after year, however, the development level of digital basic facilities in some regions (especially Zhejiang and Guangdong) in 2020 has dropped slightly brought by the influences of the COVID-19 epidemic.
(2) For integrated development performance, the annual value of utility function of integrated development in Guangdong achieves 0.9639 from 2015 to 2020, which is much higher than Jiangsu (the annual value of that is 0.6796) and Beijing (the annual value of that is 0.6494) ranking at the second and the third, respectively. The integrated development level of Shanghai, Zhejiang, Shandong and Sichuan are superior than other regions, ranking the fourth, fifth, sixth, and seventh, respectively. Consistent with the overall ranking, the inte- Through comparing the detailed performance of 31 regions from 2015 to 2020, it is obviously found that the DE development level of various regions is primarily affected by the integrated development performance, innovation ability performance, and social benefits performance. Therefore, the backward regions in DE development, especially Guangxi, Jilin, Tibet, Qinghai, Xinjiang, and Ningxia, should emphasize the development of software industry and information technology industry, increase the input of research & development funds for industrial enterprises and the average wage of urban employees in information transmission, computer services and software industries, and propose talent incentive measures to increase the employment of urban units in information transmission, software and information technology services and the number of research & development personnel in enterprises above designated size, so that the development level of DE of these regions can be greatly improved.

Discussion on the robustness of the established MCDM model
To empirically prove the effectiveness and validation of the MCDM model combining Fuzzy-Delphi, AEW, BWM and MARCOS, 4 hybrid MCDM models are chosen to be comparison models and three ranking similarity coefficients are utilized to analyze ranking similarity. 4 comparison models for analyzing ranking similarity are indicated in Table 9. The detailed evaluation process of traditional TOPSIS and VIKOR can be referred to Behzadian et al. [42] and Zhao et al. [43]. Through comparatively discussing the ranking similarity among Model 1, Model 2, and Model 3, the validation of the established integrating weighting method is proved, and by discussing the ranking similarity among Model 1, Model 4, and Model 5, the validation of the evaluation model MARCOS is proved.
where x i and y i demonstrate the sequences of rankings, and N is the number of evaluated provincial level regions in the rankings. It is important to note that in actual MCDM studies, the coherence of the top alternative is of great importance for evaluating the robustness of the evaluation results, as the top one is on Table 9. The comparative models for ranking similarity analysis.

Models
Model description

Model 1
The proposed hybrid MCDM model integrating Fuzzy-Delphi, AEW, BWM and MARCOS

PLOS ONE
behalf of the optimal selection for actual decision making. Hence, the WS similarity coefficient (WS) makes the ranking variance of the top alternative exert much more influences on final ranking similarities. In terms of the actual data of 17 sub-criteria in 31 provincial level regions in China, ranking results are calculated via 5 models from the overall performance and five sub-performances of DE development, including digital infrastructure performance, integrated development performance, social benefits performance, innovation ability performance, and electronic-commerce performance. Afterwards, similarity coefficients between ranking results of the established MCDM model and 4 comparison models are calculated via Eqs (24)- (26), listing in Table 10.
It can be discovered from Table 10 that: (1) In terms of overall performance, integrated development performance, and innovation ability performance, rankings of Model 1 and 4 comparison models are highly coherent. All similarity coefficients are 1.000 meaning that after changing the weighting approach and (2) In terms of digital infrastructure performance, social benefits performance, and electroniccommerce performance, through comparing similarity coefficients of Models 1, 2, and 3, we can discover that all similarity coefficients of these three models are 1.0000 indicating that changing the weighting method will not influence ranking results, hence, the integrated weighting method employed in this paper has better robustness in evaluating DE development. Through comparing similarity coefficients of Models 4 and 5, it can be found that all similarity coefficients of these two models are all larger than 0.95, which demonstrates that a slight distinct exist in ranking results among Models 1, 4, and 5 in digital infrastructure, social benefits, and electronic-commerce performances. Specifically, similarity coefficients values of Model 4 are larger than those of Model 5, which illustrates that ranking results of Model 4 in digital infrastructure, social benefits, and electronic-commerce performances are relatively highly similar with those of Model 1. Moreover, values of WS coefficient of Models 4 and 5 are all greater than 0.98, which means the top provincial level regions of Models 4 and 5 are highly homologous with those of Model 1 in terms of digital infrastructure, social benefits, and electronic-commerce performances. Therefore, it can be summarized that the change of comprehensive evaluation model will influence the ranking results of sub-performance to a certain degree, nevertheless the impacts are mild, that means the established MCDM model has good robustness in assessing sub-performance of DE development.
Therefore, the above ranking comparison discussion indicates that changing the weighting method exert no impact on ranking results of both overall performance and sub-performances, while changing comprehensive evaluation model will have slight impact on ranking results of sub-performances but will not affect ranking results of overall performance. Hence, the established MCDM methodology integrating Fuzzy-Delphi method, AEW method, BWM, and MARCOS model has superior robustness and validity in evaluating DE development.

Conclusions
With the deepening of a new round of scientific and technological revolution and industrial reform, digital technology with the internet as the core has been continuously innovated and widely penetrated into various economic fields, which has brought great changes in productivity and mode of production. The world has entered a new era that DE promotes economic development. In October 2020, China proposed to upgrade the DE into a national strategy, accelerate the development of DE, further accelerate digital industrialization and industrial digitization, deeply integrate the DE with the real economy, give sufficient play to the driving part of DE in economic growth as well as promote high-quality economic development. It is demonstrated that the DE has become the focus of China's economic development planning and a new engine to enhance national strength. Evaluating the development level of DE in various regions of China is conducive to timely find out the shortcomings in China's DE development, accurately grasp the key factors affecting China's economic growth, and then provide an important basis for putting forward corresponding policy suggestions to scientifically guide the direction of China's economic development. Therefore, coupled with the connotation and particular features of DE, this paper constructs the evaluation indicator system of DE development, and makes a comprehensive evaluation of the development level of DE in various regions of China.
Firstly, the initial indicator system is built from digital infrastructure, integrated development, social benefits, innovation ability, and electronic-commerce dimensions containing 31 initial sub-criteria through referring to related reports and published literature. Then, the Fuzzy-Delphi approach is applied to screen the crucial indicators considering 5 experts judgments. And the final index system is constructed containing 17 quantitative sub-criteria. The development level of DE in 31 regions in China will be evaluated. Aiming at comparatively analyzing the development of DE in different regions, actual data of 17 sub-criteria with regard to these 31 provincial level regions from 2015 to 2020 can be gathered from the website of National Bureau of Statistics. Afterwards, objective weights from 2015 to 2020 can be calculated via the AEW method based on actual data of 17 sub-criteria with regard to 31 provincial level regions from 2015 to 2020. The BWM is also applied to identify the subjective weights of 17 indicators in terms of experts' judgments on important degree of indicators. Then the integrated weights from 2015 to 2020 are computed via the basic principle of moment estimation in accordance with subjective weights determined through the BWM and the objective weights estimated via the AEW approach. The top four sub-criteria with relatively greater integrated weights are C 42 representing research & development funds for industrial enterprises above designated size with 0.0839 in 2020, C 31 representing express quantity with 0.0838 in 2020, C 23 representing express business income with 0.0746 in 2020, and C 41 representing the number of patent applications of industrial enterprises above designated size with 0.0745 in 2020. Moreover, the last three sub-criteria with relatively less integrated weights are C 13 representing long distance optical cable line length with 0.0406 in 2020, C 11 representing telephone penetration ratio with 0.0391 in 2020, and C 32 representing average wage of urban employees in information transmission, computer services and software industries with 0.0320 in 2020. From sub-performance perspectives, the integrated weights of C 2 on behalf of integrated development are all largest in different years, followed by C 4 representing innovation ability. While the integrated weights of C 1 on behalf of digital infrastructure in different years are all the smallest.
After determining the final indicator system for DE development evaluation and integrated weights of all sub-criteria, MARCOS model is applied to rank the development level of DE of 31 provincial level regions in China. Case analysis illustrates that the values of utility function in Guangdong, Jiangsu, Zhejiang, Beijing, Shandong, Shanghai, and Sichuan are much greater than other regions, among which, Guangdong, Jiangsu, Zhejiang, and Beijing always maintain in the top four from 2015 to 2020. Due to the late economic development in the southwest and northwest regions in China, the development level of the DE is obviously lower than that of other regions, among which, Xinjiang and Ningxia ranked at the last two positions, of which the average values of utility function are 0.1218 and 0.1154 from 2015 to 2020, respectively. The development level of DE in the northeast region (especially Jilin and Heilongjiang provinces) also rank at the downstream position of China. For the detailed performance of 31 regions from 2015 to 2020, Guangdong always has the best performance in digital infrastructure, integrated development, social benefits, innovation ability, and electronic-commerce performances, while the northwest regions in China are inferior to others in sub-performances. Through comparing the detailed performance of 31 regions from 2015 to 2020, we can summarize that the DE development level of various regions is primarily affected by the integrated development performance, innovation ability performance, and social benefits performance. Therefore, the backward regions in DE development, especially Guangxi, Jilin, Tibet, Qinghai, Xinjiang, and Ningxia, should emphasize the development of software industry and information technology industry, increase the input of research & development funds for industrial enterprises above designated size and the average wage of urban employees in information transmission, computer services and software industries, and propose talent incentive measures to increase the employment of urban units in information transmission, software and information technology services and the number of research & development personnel in enterprises above designated size, so that the development level of DE of these regions can be greatly improved.
The robustness of the constructed MCDM model coupling Fuzzy-Delphi, AEW, BWM and MARCOS is discussed through selecting four comparison models and utilizing three similarity coefficients of rankings including r S , r W , and WS to estimate the ranking similarity. The ranking comparison discussion indicates that changing the weighting method will not change ranking results of both overall performance and sub-performance, while changing comprehensive evaluation model will have slight impact on ranking results of sub-performance but will not affect ranking results of overall performance. Hence, the constructed MCDM methodology coupling Fuzzy-Delphi method, AEW method, BWM, and MARCOS model has superior robustness and validity in evaluating DE development.
It needs to be noted that the hybrid MCDM model established in this investigation has wide applications. It cannot only be limited to estimate the DE development level in various regions of China, but also can be applied to other countries. However, there are still some limitations of this model. If there is only one region to be estimated, the AEW approach will be unapplicable to calculate the objective weights. For one evaluated alternative, the weight determination method with variance-driven principle, entropy weighting approach, and AEW method are all not applicable [46,47]. And the MARCOS model is also inappropriate except for setting the positive and negative ideal solutions ahead of time. For one evaluated alternative issue, fuzzy theory-based evaluation method and matter element extension method are applicable for composing the MCDM model. Therefore, in the future research, it is necessary to build a comprehensive evaluation model for assessing the development level of DE suitable for different situations, and explore the driving factors of high-quality development of DE, so as to provide policy reference for high-quality development of DE.